Models of the Time Window of Integration

Evidence for multisensory integration is found in many different circumstances, notably as facilitation or inhibition of responses to cross-modal stimulus sets, compared to unimodal stimulation. Prime examples of facilitation include increased firing rate of a multisensory neuron, acceleration of manual or saccadic reaction time, or effective integration of audiovisual speech. Besides facilitation and inhibition, cross-modal stimulation may also result in no discernable effect at all, that is, beyond what would be expected from unimodal stimulation alone. This has sometimes been referred to as a situation where the cross-modal stimulus combination falls outside of a spatiotemporal “window of integration” (e.g., Meredith, 2002). This window concept has enjoyed increasing popularity over the past 5–10 years as a major determinant of the dynamics of multisensory integration at both the neural and behavioral level of observation. The purpose of this chapter is to elucidate that time window of integration is not only a metaphor describing a host of cross-modal stimulation effects in various different contexts but, rather, has become a cornerstone of a quantitative approach in reaction time modeling of multisensory integration. We will not review the extensive experimental literature on the topic here (but see Diederich & Colonius, 2008a). Instead, we first demonstrate, within the time window of integration (TWIN) model (Colonius & Diederich, 2004), how the concept leads to empirically testable predictions across different response time paradigms. Second, it is shown that the time window of integration notion can be incorporated into current models of multisensory integration based on principles of optimal statistical inference (see Ernst, 2006 for a review). All along, we discuss open questions including possible extensions of this approach beyond reaction time analysis. Although a window of integration has been defined for both spatial and temporal aspects of a cross-modal stimulus set (e.g., Wallace et al., 2004) and has even been suggested for higher-level aspects such as semantic congruity (e.g., van Attefeldt, Formisano, Blomert, & Goebel, 2007), we confine this discussion to the temporal dimension within the reaction time context considered here. On a descriptive level, the time window hypothesis holds that information from different sensory modalities must not be too far apart in time so that integration into a multisensory (perceptual) unit may occur. Thus, an infinitely large time window will lead to mandatory integration, whereas a zero-width time window will rule out integration entirely. In order to have a nontrivial situation, the time window must take on some finite value. For example, in audiovisual speech perception, auditory speech has to lag behind matching visual speech, that is, lip movements, by more than 250 msec for any asynchrony to be perceived (Conrey & Pisoni, 2006; Dixon & Spitz, 1980; van Wassenhove, Grant, & Poeppel, 2007). When a sensory event simultaneously produces both sound and light, we usually do not notice any temporal disparity between the two sensory inputs (within a distance of up to 20–26 m), even though the sound arrives with a small delay, a phenomenon referred to as “temporal ventriloquism” (cf. Lewald & Guski, 2004; Morein-Zamir, Soto-Faraco, & Kingstone, 2003; Spence & Squire, 2003). In orienting responses toward a visual-auditory stimulus complex, acceleration of (saccadic) reaction times has been observed under a multitude of experimental settings within a time window of 150 to 250 msec (e.g., Corneil, Van Wanrooij, Munoz, & Van Opstal, 2002; Diederich & Colonius, 2004a, 2008a, 2008b; Diederich & Colonius, 2009; Frens, Van Opstal, & Van der Willigen, 1995; Navarra, Hartcher-O’Brien, Piazza, & Spence, 2009; Navarra et al., 2005; Romei, Murray, Merabet, & Thut, 2007; Van Opstal & Munoz, 2004; van Wanrooij, Bremen, & Van Opstal, 2009). On the neural level, a temporal window of integration has been observed at multisensory convergence sites such as the superior colliculus in cat and other animals: enhanced spike response rates of multisensory neurons occur when periods of unimodal peak activity overlap within a certain time range (King & Palmer, 1985; Meredith, Nemitz, & Stein, 1987; Rowland & Stein, 2008). Note that the fact that estimates of the width of the time window differ widely, ranging from 40 to 600 msec, is not really surprising given that these estimates arise from rather different